Extending CLIP for Category-to-Image Retrieval in E-Commerce

E-commerce provides rich multimodal data that is barely leveraged in practice. One aspect of this a category tree being used search and recommendation. However, practice, during user’s session there often mismatch between textual visual representation given category. Motivated by the problem, we introduce task category-to-image retrieval e-commerce propose model for task, CLIP-ITA. The leverages information from multiple modalities (textual, visual, attribute modality) to create product representations. We explore how adding impacts model’s performance. In particular, observe CLIP-ITA significantly outperforms comparable only modality modality.